f DEPARTMENT  OF  DEFENCE 

DEFENCE  SCIENCE  &  TECHNOLOGY  ORGANISATION 


DSTO 


An  Approach  for  Identifying  and 
Characterising  Problems  in  the 
Iterative  Development  of  C3I 
Capability 

Gina  Kingston,  Derek  Henderson 
and  Rudi  Vernik 

DSTO-TN-0238 


Pktribotion  statement  a 

tor  Public  Release 
Distribution  Unlimited 


20000331  054 


An  Approach  for  Identifying  and  Characterising  Problems 
in  the  Iterative  Development  of  C3I  Capability 


Gina  Kingston,  Derek  Henderson  and  Rudi  Vernik 

Information  Technology  Division 
Electronics  and  Surveillance  Research  Laboratory 

DSTO-TN-0238 


ABSTRACT 

This  report  presents  a  research  approach  that  has  been  used  to  determine  the  key  problems  faced 
by  the  Australian  Defence  Organisation  (ADO)  in  the  iterative  development  of  C3I  capability. 
Repeated  application  of  the  approach,  combined  with  knowledge  of  strategic  and  technological 
changes,  could  be  used  to  assess  process  improvement  activities. 

The  approach  uses  interviews  of  stakeholders  in  the  Iterative  Capability  Development  (ICD) 
process  to  gather  information  about  the  perceived  problems.  As  well  as  documenting  the 
approach  used  in  a  study  of  the  key  problems  with  the  ICD  process,  this  report  illustrates  the 
application  of  study  design  principles,  such  as  methods  for  minimising  potential  biases  in  an 
interview-based  study. 
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An  Approach  for  Identifying  and  Characterising 
Problems  in  the  Iterative  Development  of  C3I 

Capability 

Executive  Summary 

This  report  presents  a  research  approach  that  has  been  used  to  determine  the  key  problems  faced 
by  the  Australian  Defence  Organisation  (ADO)  in  the  iterative  development  of  C3I  capability. 

The  approach  used  interviews  of  the  key  stakeholders  in  the  Iterative  Capability  Development 
(ICD)  process  to  gather  information  about  the  perceived  problems.  The  interviews  consisted  of 
four  main  activities:  an  overview,  evaluation  of  a  model  of  the  ICD  process,  problem 
identification  and  a  problem  elaboration  phase.  The  interview  process  was  constructed  to  reduce 
the  possibility  of  biases.  To  this  end,  the  elaboration  of  problems  was  supported  by  a 
questionnaire.  Other  measures  adopted  to  improve  the  rigour  of  the  study  included: 

•  Early  identification  of  the  criteria  used  to  assess  the  problems. 

•  Identification  of  key  stakeholders  in  the  ICD  process  and  careful  selection  of  study 
participants. 

•  Use  of  an  interview  process  that  included  several  avenues  for  verifying  participants’ 
contributions,  such  as  requiring  detailed  information  and  examples  of  the  problems. 

•  Use  of  tool  support  to  document  and  confirm  problems  as  they  were  discussed. 

•  Confirmation  and  clarification  of  the  information  recorded. 

•  Early  evaluation  of  the  approach  during  interviews  with  DSTO  personnel. 

The  report  also  discusses  insights  into  the  approach  obtained  during  its  successful  application. 
Sixteen  participants  in  the  ICD  process  were  interviewed,  including  the  entire  C3I/IM 
Development  Committee  and  three  project  managers.  In  particular,  the  report  discusses  the 
importance  of  evaluating  not  only  the  approach  but  also  the  implementation  of  the  approach  by 
the  research  team. 

It  is  hoped  that  other  researchers  can  benefit  from  this  report  and  the  detailed  discussion  of 
methods  used  to  minimise  the  potential  biases  in  an  interview-based  study. 
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1.  Introduction 


This  report  describes  an  approach  that  has  been  used  to  identify  and  characterise  problems  in  the 
Iterative  Capability  Development  (ICD)  of  C3I. 

The  approach  was  designed  around  interviews  with  stakeholders  in  the  Australian  Defence 
Organisation  (ADO).  The  approach  was  used  to  identify  the  stakeholders’  perceptions  of  current 
problems  in  the  ICD  process  and  the  impact  of  these  problems  on  the  delivery  of  C3I  capability 
to  the  Australian  Defence  Force  (ADF).  The  objectives  of  the  study  were  to  document  these 
problems,  to  analyse  the  impact  of  the  problems,  and  to  identify  areas  of  further  research  that 
have  the  potential  to  improve  the  ICD  process.  It  is  believed  that  the  repeated  application  of  the 
approach,  combined  with  knowledge  of  strategic  and  technological  changes,  could  be  used  to 
evaluate  the  impact  of  process  improvement  activities.  (See  Section  6). 

The  report  includes  a  brief  overview  of  the  ICD  process  (Section  2),  the  research  objectives  and 
a  summary  of  the  approach  (Section  3),  and  a  discussion  of  the  research  method  (Section  4).  For 
simplicity,  these  sections  discuss  the  design  of  the  approach  in  terms  of  the  single  study  already 
completed.  The  results  of  this  study  are  reported  separately  in  (Henderson  and  Kingston  1999). 
It  is  believed  that  the  credibility  of  these  results  is  enhanced  through  the  rigorous  approach  that 
was  used  and  which  is  described  in  this  report. 

This  document  may  also  be  useful  to  researchers  conducting  similar  studies  in  other  areas,  as 
this  document  discusses  the  rationale  behind  the  approach.  Section  4  discusses  how  potential 
biases  were  addressed  in  the  design  of  the  approach.  Section  5  discusses  lessons  learned  while 
conducting  the  interviews,  and  Section  6  discusses  the  implications  for  other  researchers. 
However,  researchers  are  advised  to  also  consult  other  texts;  this  report  is  not  intended  as  a 
tutorial  on  empirical  techniques  and  it  does  not  provide  a  detailed  discussion  of  alternative 
approaches.  Ellis  (1994)  provides  a  well-structured,  easy  to  read,  introduction  to  empirical 
techniques  in  general.  Yin’s  book  (1994)  on  case  studies  is  worth  reading,  and  Denzin  (1994) 
provides  a  more  detailed  collection  of  papers  on  specific  techniques,  such  as  observational 
techniques  (Adler  and  Adler  1994)  and  interviews  (Fontana  and  Frey  1994). 

The  results  of  studies  involving  interviews  often  lack  credibility  because  of  the  subjective  nature 
of  the  participants’  contributions,  the  ad  hoc  manner  in  which  participants  are  selected,  and 
because  the  views  of  the  researchers  can  influence  the  participants.  In  order  to  address  these 
criticisms,  the  approach  was  designed  with  the  following  features; 

•  Identification  of  key  stakeholders  in  the  ICD  process  and  careful  selection  of  study 
participants. 

•  Early  identification  of  the  criteria  used  to  assess  the  problems. 

•  Use  of  an  interview  process  that  included  several  avenues  for  verifying  participants’ 
contributions,  such  as  requiring  detailed  information  and  examples  of  the  problems. 

•  Use  of  tool  support  to  document  and  confirm  problems  as  they  were  discussed,  and  to  later 
analyse  problems. 

•  Confirmation  and  clarification  of  the  information  recorded. 

•  Early  evaluation  of  the  approach  during  interviews  with  DSTO  personnel. 
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The  design  of  the  approach  had  to  balance  the  desire  to  use  a  rigorous,  scientific,  approach;  the 
desire  for  flexibility  in  the  interview  process;  and  awareness  of  interviewees’  time  constraints. 
Thus,  the  research  approach  has  some  limitations  when  considered  fi-om  any  one  of  these 
viewpoints.  For  example,  the  selection  of  participants  was  not  rigorously  defined  before  the 
interviews;  it  was  modified  after  the  initial  interviews.  The  participants  tended  to  focus  on 
recent  problems  and  did  not  always  constrain  their  responses  to  discussions  of  first  hand 
experiences.  Nevertheless,  we  believe  that,  because  this  approach  was  used,  the  results  of  the 
study  are  more  credible  than  could  have  been  achieved  with  an  ad  hoc  series  of  interviews. 

2.  Iterative  Capability  Development  (ICD) 

This  section  describes  the  ICD  domain  for  which  the  approach  was  developed.  The  term  ICD 
was  devised  by  the  researchers  to  encapsulate  all  the  processes  used  by  Defence  to  plan, 
develop,  acquire,  maintain  and  evolve  C3I  capability  in  an  iterative  manner.  It  includes,  but  is 
not  limited  to  the  high  level  conceptual  planning  that  occurs  in  C3I  Capability  Development 
Division,  the  acquisition  processes  used  by  the  Defence  Acquisition  Organisation  (DAO),  and 
the  processes  used  by  the  support  organisations  to  maintain  C3I  capability. 

Defence  has  gradually  adopted  ICD  over  the  past  few  years.  Previously,  development  was 
accomplished  using  the  traditional  or  ‘Big-Bang’  approach.  In  the  traditional  approach,  the 
phases  are  conducted  in  a  linear  manner;  requirements  are  set  at  the  start  of  an  acquisition;  and 
there  are  few  feedback  loops.  For  the  development  of  C3I  capability,  iterative  approaches  that 
incorporate  feedback  loops  are  generally  preferable  to  the  traditional  approach  for  two  main 
reasons.  First,  C3I  systems  need  to  be  adaptable  to  cope  with  rapid  changes  in  the  strategic 
environment,  and  subsequent  changes  to  operational  requirements.  Second,  the  effectiveness  of 
C3I  systems  is  dependent  on  the  use  of  modem  information  technology  (IT),  which  is  also 
subject  to  rapid  change.  The  traditional  approach  does  not  have  the  flexibility  to  cope  with  this 
high  rate-of-change. 

Figure  1  shows  the  various  organisations  that  contribute  to  the  IGD  process  and  the  main 
interactions  between  the  organisations.  This  diagram  documents  the  researchers’  understanding 
of  the  ICD  process  before  any  interviews  were  conducted.  It  was  used  as  a  discussion  point  at 
the  start  of  the  interviews  and  provided  a  fi-amework  for  recording  the  interviewees’  knowledge 
of  the  ICD  process.  The  main  limitation  of  the  model  is  that  it  only  represents  the  DAO 
perspective  and  does  not  include  a  general  Defence  Information  Systems  Group  (DISG) 
perspective.  Furthermore,  the  Joint  Support  Agency  (JSSA)  was  not  part  of  the  DISG  when  the 
model  was  developed. 
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Figure  1.  Iterative  Capability  Development. 
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Figure  2.  EA  -  an  iterative  approach 


Evolutionary  acquisition  (EA)  is  central  to  ICD.  In  recent  years,  the  DAO  formally  introduced 
EA  and  it  is  now  used  on  a  several  projects  including  the  major  command  support  systems 
(CSS)  and  a  number  of  Communications  projects.  The  main  thrust  of  EA,  as  illustrated  in 
Figure  2,  is  the  incremental  specification,  design,  implementation,  test,  delivery,  operation  and 
maintenance  of  systems.  Delivery  of  each  incremental  release  increases  the  overall  capability  of 
the  system.  In  this  way,  users  of  the  system  get  early  access  to  functionality  and  are  encouraged 
to  provide  feedback  on  functionality  and  performance.  The  feedback  is  used  in  subsequent 
increments  to  shape  the  development  of  the  system  as  it  evolves  to  its  final  form.  See 
(Henderson  and  Gabb  1997)  for  a  detailed  discussion  of  EA. 
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EA  is  only  one  aspect  of  the  ICD  process,  albeit  an  important  one.  EA  involves  a  fundamental 
shift  in  the  way  that  Defence  acquires  equipment;  one  that  requires  changes  not  only  to  the 
DAO,  but  also  to  the  other  processes  used  in  the  development  of  C3I  capability.  Many  of  the 
processes  required  for  successful  EA  are  different  to  those  used  on  traditional  projects  (Defence 
Capability  Committee  1998).  For  example,  the  project  approval  process  used  in  a  traditional 
project  is  not  suitable  for  an  EA  project.  Departments  within  the  ADO  that  contribute  to  the  ICD 
process  include,  but  are  not  limited  to: 

•  C3I  Capability  Development  Division,  who  develop  capability  development  proposals  for 
consideration  by  the  Defence  Capability  Committee  (DCC). 

•  The  Defence  Capability  Committee,  which  approves  an  acquisition. 

•  The  DAO  project  offices,  which  are  responsible  for  acquiring  a  system  within  the  guidelines 
established  by  the  DCC. 

•  The  users,  who  request,  use  and  evaluate  the  acquired  capability. 

•  DSTO,  who  provides  input  to  the  process  through  advice,  research,  and  the  development  of 
Capability  Technology  Demonstrators. 

While  the  approach  has  focused  on  ICD  in  the  ADO,  industry  also  has  a  key  role  in  the  ICD 
process. 

It  is  inevitable  that  there  will  be  problems  in  an  activity  involving  so  many  departments  and  so 
many  new  processes.  The  aim  of  the  approach  was  to  enable  those  problems  to  be  identified, 
and  their  impact  to  be  qualified  and  quantified. 

3.  Approach 

This  section  summarises  the  approach  designed  to  investigate  problems  in  the  ICD  of  C3I. 
Section  3.1  discusses  the  objectives  of  the  approach.  Section  3.2  outlines  the  activities 
conducted,  and  Section  3.3  discusses  the  criteria  used  to  evaluate  problems  in  the  ICD  of  C3I. 
Section  4  discusses  the  design  of  the  approach  in  more  detail,  focusing  on  how  the  design 
reduces  the  likelihood  of  biases  affecting  the  results  of  studies  that  use  the  approach. 


3.1  Objectives 

The  research  approach  was  designed  to  meet  two  main  objectives.  The  first  objective,  Oi  was  to 
identify  problems  that  would  benefit  firom  further  research.  As  shown  in  Figure  5,  a  two-step 
approach  was  used  to  identify  research  problems.  The  first  step  was  the  identification  of  the 
problems  encountered  by  Defence  personnel  involved  in  the  ICD  process.  The  second  step  was 
the  characterisation  of  these  problems.  This  not  only  allowed  the  identification  of  research 
problems,  but  also  provided  a  baseline  against  which  the  results  of  process  improvement 
activities  can  be  assessed.  The  second  objective,  O2,  was  to  evaluate  and  refine  the  model  of  the 
ICD  process  given  in  Figure  1. 
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Figure  3:  Steps  in  identifying  problems  that  would  benefit  from  research. 

3.2  Outline  of  the  Approach 

This  section  outlines  the  research  approach,  as  it  was  implemented  in  the  completed  study.  The 
research  approach  was  based  around  interviews  with  key  stakeholders  in  the  ICD  process.  As 
shown  in  Figure  4,  the  interviews  form  only  part  of  the  approach;  additional  activities  were 
conducted  before  and  after  the  interviews. 


Before  the  interviews,  the  participants  were  supplied  with  information  to  allow  them  to  prepare 
for  the  interviews.  It  provided  background  material  and  suggested  activities  that  they  might 
conduct  to  prepare  for  the  review.  For  example,  they  could  compile  a  log  of  problems 
encountered  during  the  week  prior  to  the  interview. 

The  interview  itself  consisted  of  the  following  four  steps: 

1 .  Overview.  The  researchers  introduced  themselves  and  explained  the  interview  process. 

2.  Evaluate  Process  Model.  The  ICD  process  model  (Figure  1)  was  discussed  and  the 
interviewee’s  role  in  the  process  was  identified  and  discussed.  This  step  had  two  purposes: 
it  addressed  objective  O2,  as  it  allows  deficiencies  in  the  model  to  be  noted  and  used  to 
refine  the  model;  and  it  was  a  natural  introduction  to  the  next  activities,  which  related  to 
objective  Oi. 

3.  Problem  Identification.  Once  the  participants  had  started  talking  about  their  role  in  the 
ICD  process,  and  the  interactions  they  have  with  the  other  entities  in  Figure  1,  the 
participants  were  encouraged  to  discuss  the  problems  that  they  had  encountered  in  that  role. 
Most  participants  required  little  or  no  encouragement  to  discuss  the  problems  they  had 
encountered. 

4.  Problem  Evaluation.  After  problems  had  been  identified,  they  were  assessed  against  the 
research  criteria  discussed  in  Section  3.3. 
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Figure  4:  Activities  involving  the  interview  participants. 

After  the  interview,  the  participants  were  given  the  opportunity  to  provide  additional 
information  and  to  check  that  the  information  recorded  during  the  interview  accurately  reflected 
their  opinions  and  experiences. 


3.3  Identifying  and  Characterising  Potential  Research  Areas 

The  information  used  to  assess  the  problems  was  based  on  seven  criteria  that  should  drive  the 
selection  of  ICD  research  problems,  and  the  research  approach  used  to  address  these  problems. 
These  criteria  are  summarised  in  Table  1 . 
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Table  1:  Research  Criteria 


Criterion 

Description 

Problem  Areas 

Cl. 

Perceived  as  a  problem  by  the  stakeholders. 

C2. 

Impact  on  the  iterative  capability  development  process. 

C3. 

Potential  impact  of  solutions. 

Research  Approaeh 

C4. 

Able  to  be  empirically  evaluated,  using  a  scientific  approach. 

C5. 

Able  to  be  evaluated  within  a  reasonably  short  time  frame  (3-6  months). 

C6. 

Able  to  control  or  manipulate  problem  and  solutions  for  evaluation. 

Cl. 

Able  to  be  effectively  and  efficiently  evaluated  by  the  researchers. 

The  first  three  criteria  are  concerned  with  the  potential  improvement  to  the  ICD  process.  As 
shown  in  Figure  5,  there  should  be  a  problem,  it  should  impact  the  ICD  process,  and  it  should  be 
possible  to  address  the  problem.  (Note  that  the  approach  described  in  this  report  focuses  on 
perceived  problems,  rather  than  on  uncovering  hidden  problems.)  The  next  three  criteria 
consider  the  approaches  that  can  be  used  to  research  the  problem.  In  particular,  they  are 
concerned  with  the  susceptibility  of  the  problem  to  empirical  investigation.  The  final  criterion  is 
concerned  with  the  suitability  of  the  problem  for  research  by  a  particular  research  team. 


The  seven  criteria  are  now  described  in  more  detail. 


Cl. The  key  stakeholders  in  the  ICD  process  should  perceive  the  problems  as  relevant.  This 
criterion  is  required  to  ensure  that  the  future  ICD  research  is  of  interest  to,  and  is  relevant  to. 
Defence.  It  would  be  difficult  to  justify,  and  receive  support  for,  work  that  was  not  deemed 
important  by  the  key  stakeholders.  For  each  potential  problem  identified  and  recommended 
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for  further  study,  the  researchers  should  aim  to  determine  whether  the  problem  is  real  or 
perceived. 

C2.  The  problems  should  have  a  significant  potential  impact  on  ICD  process.  Problems  might 
affect  a  project’s  cost,  its  schedule,  the  quality  of  the  delivered  system,  and  the  relationship 
between  the  contractor  and  the  ADO.  Alternatively,  problems  might  be  irritating  and 
annoying,  but  have  little  impact  of  the  outcomes  of  the  ICD  process. 

C3.  It  should  be  possible  to  identify  potential  solutions,  or  partial  solutions,  to  the  problem  that 
would  have  a  high  impact  on  the  perceived  efficiency  or  effectiveness  of  ICD.  The  key 
stakeholders  might  be  able  to  identify  potential  solutions.  Alternatively,  the  nature  of  the 
problem  might  suggest  avenues  for  determining  a  solution. 

C4.  The  research  approach  used  to  investigate  the  problems  will  depend  on  whether  or  not  the 
problems  and  their  potential  solutions  are  amenable  to  rigorous  empirical  evaluation.  To  be 
amenable  to  empirical  investigation,  the  scope  of  the  problems  and  potential  solutions 
should  be  clearly  identified  and  the  outcomes  of  changes  to  the  process  should  be 
measurable.  These  measures  might  be  absolute  -  such  as  the  number  of  problems  reported 
during  the  first  year  of  operation  -  or  based  on  subjective  evaluations  -  such  as  the  result  of 
a  survey  of  users.  Some  problems  will  require  additional  theoretical  and  practical 
investigation  as  well  as,  or  instead  of,  empirical  investigation. 

C5.  The  timescale  for  conducting  the  research  should  be  considered.  The  ICD  process  is 
undergoing  rapid  change,  so  for  many  research  programs  (empirical  or  otherwise)  the 
timing  of  the  research  is  a  critical  factor.  Furthermore,  the  potential  benefit  of  any  research 
program  to  existing  projects  may  need  to  be  considered. 

C6.  If  an  empirical  investigation  is  being  considered,  it  should  be  possible  to  manipulate  the 
ICD  process  to  assess  the  impact  of  potential  solutions.  That  is,  ideally,  it  is  possible  to 
compare  the  usual  problem  with  the  proposed  solution,  to  determine  whether  or  not  the 
proposed  solution  does  improve  the  ICD  process. 

Cl.  The  researchers  should  have  the  relevant  experience  and  expertise  to  address  the  problem. 

The  objective  of  the  approach  described  in  this  report  is  to  enable  information  that  can  be  used 
to  assess  the  first  six  criteria  to  be  captured.  The  results  of  the  study,  as  reported  in  (Henderson 
and  Kingston  1999),  indicate  the  relative  scores  of  the  problems  against  the  first  criteria, 
discusses  the  other  criteria,  and  identifies  areas  where  further  study  is  desirable.  Section  4.3 
describes  how  the  criteria  were  assessed.  It  should  be  noted  that  problems  did  not  need  to  score 
highly  in  all  seven  areas  to  be  selected.  Indeed,  it  would  be  surprising  if  any  problems  did  score 
highly  in  all  seven  areas.  Section  4  now  considers  the  design  of  the  approach  in  more  detail. 

4.  Research  Method 


Section  3.2  discussed  how  the  approach  consisted  of  interviews  with  key  stakeholders  in  the 
ICD  process.  This  approach  was  chosen  to  meet  the  first  objective  of  identifying  areas  where 
further  research  is  desirable.  As  discussed  in  Section  3.3,  the  views  of  the  key  stakeholders  were 
one  of  the  criteria  used  to  assess  whether  or  not  problems  should  be  researched.  Other 
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approaches,  such  as  the  use  of  case  studies,  experiments,  or  fixed  questionnaires  were  not 
believed  to  be  suitable  as  it  was  anticipated  that  stakeholders  from  different  parts  of  the  ICD 
process  would  encounter  different  problems.  A  case  study  detailed  enough  to  identify  the 
problems  and  broad  enough  to  cover  all  the  areas  was  not  feasible.  An  experiment  or  a  rigid 
questionnaire  could  not  be  used  because  the  problems  were  not  known  in  advance. 

However,  an  interview-based  research  design  has  several  potential  problems.  The  results  of  such 
studies  can  be  biased  because  the  views  of  stakeholders  who  are  interviewed  do  not  reflect  the 
views  of  all  the  stakeholders;  because  of  limitations  in  the  participants  ability  to  provide 
unbiased  information;  because  the  participants  do  not  have  all  the  information;  or  because  the 
researchers  influence  either  the  information  provided  by  the  participants,  or  its  interpretation. 

The  approach  used  to  investigate  problems  in  the  ICD  process  was  designed  to  limit  the  impact 
of  these  potential  sources  of  bias.  Section  4.1  discusses  the  selection  of  participants,  and  how 
the  problem  of  selecting  a  representative  sample  was  addressed.  Section  4.2  discusses  how  the 
potential  biases  of  the  researchers  were  addressed  in  the  design  of  the  questionnaire.  However, 
Section  4.3  discusses  how  the  detailed  construction  of  the  questionnaire  was  influenced  more  by 
the  potential  biases  of  the  participants,  which  also  affected  the  manner  in  which  the  interviews 
were  conducted  (Section  4.4).  Finally,  Section  4.5  describes  the  evaluation  of  the  interview 
process. 

4.1  Participants 

The  results  of  a  study  will  be  biased  if  the  interviewees  do  not  form  a  representative  sample 
from  the  population  of  interest.  As  described  in  Section  2,  the  ICD  process  involves  participants 
from  at  least  six  groups  -  Planning,  Acquisition,  Industry,  Users,  Support  Organisations  and 
DSTO  -  each  of  which  consists  of  a  large  body  of  people.  Therefore,  the  population  of  interest 
is  reasonably  large.  Furthermore,  the  researchers  did  not,  and  still  do  not,  have  an  accurate 
record  of  all  the  personnel  involved  in  the  ICD  process,  even  with  the  restriction  to  those  ADO 
personnel  involved  in  the  acquisition  of  C3I  capability. 

Therefore,  the  approach  used  in  the  study  was  to  combine  representative  samples  from  clearly 
identified  groups  with  a  series  of  interviews  selected  using  a  more  ad  hoc  approach.  At  the  start 
of  the  study,  one  clearly  defined  group  of  ICD  personnel  were  identified,  the  managers  of  C3I 
projects.  These  managers  covered  approximately  20  projects,  or  sub-projects,  that  vary 
according  to  their  branch  or  capability  area  (Communications  or  Command  Support  Systems 
(CSS))  and  the  stage  of  the  projects  (pre-contract,  post-contract  or  support).  Consequently,  it 
was  decided  to  interview  three  project  managers  of  major  projects,  who  covered  all  of  these 
areas.  The  managers  of  a  pre-contract  Communications  project,  a  post-contract  CSS  project  and 
a  support  project  were  selected. 

It  was  decided  to  supplement  the  interviews  with  these  participants,  with  interviews  of  DSTO 
personnel  and  personnel  recommended  by  the  project  managers.  The  study  was  to  clearly 
indicate  that  the  results  related  to  these  participants  were  ad  hoc  and  that  they  should  be  treated 
with  caution.  However,  during  the  course  of  the  interviews,  another  well-defined  group  of 
participants  was  identified.  This  group  is  the  C3I/IM  Development  Committee,  more  commonly 
referred  to  as  the  Gang  of  7  or  the  G7.  Therefore,  the  list  of  interviewees  was  extended  to 
interview  all  of  the  G7  and  to  clearly  identify  their  contribution  to  the  results  of  the  study. 
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Thus,  three  groups  of  participants  were  identified  with  different  properties  and  the 
corresponding  results  should  be  treated  differently.  The  results  from  the  G7  can  be  treated  with 
more  confidence  than  the  results  in  the  other  sections.  These  results  should  reflect  the  opinions 
of  the  G7  because  the  entire  sub-population  was  sampled.  The  results  from  the  project  managers 
can  also  be  treated  with  confidence.  However,  the  list  should  not  be  assumed  to  be  complete,  or 
the  ranking  applicable  to  all  projects,  because  the  entire  population  was  not  sampled.  The  final 
group  of  results  should  be  treated  with  caution,  because  the  participants  were  selected  in  an  a  J 
hoc  manner.  They  were  included  to  give  breadth  to  the  study,  and  not  to  strengthen  its  validity. 

4.2  Researchers  and  the  design  of  the  questionnaire 

Researchers  have  the  potential  to  bias  the  results  of  the  interview-based  studies  in  two  ways. 
First,  the  researchers  might  influence  the  study  participants  identification  of  problems  or  the 
discussion  of  these  problems.  Second,  even  if  the  information  discussed  during  the  interview  is 
not  biased,  the  researchers  can  bias  the  results  of  the  study  by  the  way  that  they  interpret  and 
encode  the  information. 

The  study  used  several  measures  to  reduce  the  likelihood  of  biases  being  introduced  by  the 
researchers.  The  main  measure,  which  is  discussed  in  this  section,  was  to  use  a  semi-stmctured 
interview  process.  This  involved  identification  of  the  problem  areas  by  the  study  participants 
and  the  use  of  a  standard  questionnaire  to  ensure  that  consistent  information  was  collected  about 
each  problem.  A  second  measure  was  the  use  of  two  researchers,  as  recommended  by  Adler 
(1994).  The  researchers  discussed  the  interview  and  reconciled  their  interpretations  of  the 
discussions  immediately  after  the  interviews.  The  researchers  can  also  tried  to  ensure  that 
leading  questions  were  not  used.  Other  methods  for  reducing  biases  are  discussed  in  Section  4.4. 

Interviews  can  be  structured,  unstructured  and  semi-structured.  A  structured  interview  consists 
of  a  series  of  well-defined  questions  that  are  asked  in  a  particular  order.  In  a  structured 
interview,  all  responses  and  discussions  must  fit  within  a  pre-defined  framework;  there  is  no 
opportunity  to  explore  anticipated  avenues.  When  conducting  a  structured  interview,  it  can 
sometimes  appear  that  the  researchers  are  not  listening  to,  or  are  not  responsive  to,  the  study 
participants.  Opportunities  to  capture  important  information  can  be  lost.  In  contrast,  an 
unstructured  interview  has  no  predefined  structure,  and  the  direction  of  discussions  depends 
almost  entirely  on  previous  discussions.  This  can  result  in  the  collection  of  interesting 
information.  However,  it  is  difficult  to  collect  consistent  information  from  all  the  interviewees 
using  this  approach.  Consequently,  a  semi-structured  process,  which  has  elements  of  both  the 
structured  and  unstructured  approaches,  was  used. 

In  a  structured  or  a  semi-structured  interview  process,  the  researchers  can  use  questions  with 
either  open-ended  or  closed-ended  response  options,  while  only  questions  with  open-ended 
response  options  are  normally  used  in  an  unstructured  interview  process.  Closed-ended 
questions  are  those  for  which  a  limited  set  of  responses  are  possible;  while  the  response  options 
for  open-ended  questions  are  not  limited.  For  example,  a  question  concerning  the  name  of  a 
person  or  project  normally  uses  open-ended  response  options,  and  a  question  concerning  the 
duration  of  a  project  may  use  either  closed-ended  response  options,  where  the  interviewee  has  to 
select  a  duration  from  a  list  of  options,  or  open-ended  response  options. 

Empirical  scientists  generally  prefer  closed-ended  response  options,  because  these  response 
options  require  less  coding  and  are  less  subject  to  biases  in  interpretation  (Ellis  1994).  However, 
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closed-ended  questions  cannot  be  used  to  identity  areas  that  have  not  previously  been 
considered  by  the  researchers.  This  limitation  means  the  study  of  problems  in  the  ICD  process 
should  not  rely  on  questions  with  close-ended  response  options.  That  is,  the  approach  used 
should  enable  the  identification  of  all  problems  encountered  by  the  ADO  in  the  development  of 
C3I  capability;  the  problems  should  not  be  limited  to  those  foreseen  by  the  researchers. 

However,  if  the  approach  relied  solely  on  questions  with  open-ended  response  options,  then  the 
likelihood  of  the  researchers  introducing  unintentional  biases  during  the  coding  of  the 
discussion  would  be  high.  Therefore,  the  selected  approach  was  to  use  both  open  and  closed- 
ended  response  options.  First,  discussions  with  the  participants  were  used  to  identity  problems 
that  they  had  encountered.  To  ensure  that  all  the  participants  considered  the  same  problem 
domain,  the  discussions  focused  on  the  participant’s  role  in  the  ICD  process  (Figure  1).  Second, 
for  each  problem,  the  participants  were  asked  to  answer  a  series  of  more  specific  questions 
concerning  the  problem.  Many  of  these  questions  used  closed-ended  response  options.  In 
particular,  the  information  used  to  rzuik  the  problems  was  obtained  using  questions  with  closed- 
ended  response  options. 

The  next  section  discusses  the  specific  questions  used  to  evaluate  the  problems.  It  discusses  how 
the  researchers  coded  the  information  when  the  participants  were  not  able  to  select  a  response 
from  the  pre-defined  list. 

4.3  Variables  and  Analysis  Approach 

The  questions  used  in  an  interview  need  to  be  carefully  designed  to  avoid  inaccuracies  and 
biases  that  arise  fi"om  the  participants  in  the  study.  Ellis  (1994)  identifies  four  possible  sources 
of  inaccurate  information:  1)  failure  to  understand  a  question,  2)  failure  to  recall  information,  3) 
indecision  and  4)  dishonesty.  The  approach  that  is  recommended  in  this  report  addresses  the 
first  problem  by  using  semi-structured  interviews  rather  than  a  structured  interview  or  a  survey 
approach.  Discussion  of  the  second  and  third  problems  is  deferred  until  Section  4.4.  This  section 
discusses  how  the  final  problem  was  addressed  during  the  design  of  the  questionnaire. 

The  problem  of  dishonest  information  occurs  when  the  participants  either  deliberately  or 
unintentionally  provide  false  or  misleading  information.  For  example,  the  participants  in 
surveys  and  interviews  are  often  reluctant  to  provide  information  that  they  believe  could 
adversely  affect  their  image  or  reputation.  Sometimes  participants  say  what  they  believe  the 
interviewer  wants  to  hear.  Very  occasionally,  participants  will  provide  information  intended  to 
sabotage  the  results  of  a  study.  While  these  problems  cannot  be  avoided,  they  can  be  limited  by 
requiring  the  participants  to  provide  detailed  information  about  any  area  that  they  discuss.  The 
approach  used  in  the  study  of  problems  in  the  ICD  process  was  to  ask  the  participants  to  identify 
problems  that  they  had  personally  encountered.  The  participants  were  asked  to  provide 
additional  information,  to  minimise  the  likelihood  that  any  area  identified  was  derived  from 
second-hand  information,  or  was  outside  the  participant’s  area  of  expertise.  They  were  asked  to 
describe  the  problem,  to  provide  examples  of  the  problem,  and  to  qualify  and  quantify  the 
problem. 


11 


DSTO-TN-0238 


Table  2:  Assessment  Technique. 


Criterion 

Assessment 

Cl. 

Quantitative 

C2. 

Quantitative 

C3. 

Qualitative 

C4. 

Qualitative 

C5. 

Quantitative 

C6. 

Qualitative 

Cl. 

Not  assessed 

Consequently,  the  data  collected  during  the  interviews  consisted  of  qualitative  data  collected 
during  discussions  with  the  survey  participants,  and  quantitative  data  collected  using  the 
questionnaire.  Table  2  shows  the  assessment  techniques  used  for  each  of  the  criteria  discussed 
in  Section  3.3. 

During  the  discussions  the  participants  were  encouraged  to  describe  their  role  in  the  ICD 
process  and  any  difficulties  or  problems  that  they  had  encountered.  From  this  discussion,  a  list 
of  problem  areas  was  identified  by  the  researchers,  and  confirmed  with  the  interviewees. 
Qualitative  information  about  each  problem  area  was  obtained  fi'om  additional  discussions 
focused  around  open-ended  questions  documented  in  a  questionnaire.  These  questions  were 
used  to  collect  consistent  information  about  each  problem,  including  descriptions  of  the  problem 
area,  examples  of  the  problem,  the  impact  of  the  problem  and  potential  solutions  to  the  problem. 
This  information  was  used  to  assess  the  qualitative  items  given  in  Table  3. 


Table  3:  Qualitative  Assessment 


Criterion 

Assessment  Technique 

C3. 

The  key  stakeholders  were  asked  to  identify  potential  solutions  to  the  problem. 
The  researchers  should  also  consider  potential  solutions  after  the  interviews. 

C4. 

The  researchers  used  the  description  of  the  problem  and  the  potential  solutions 
to  determine  whether  or  not  it  is  susceptible  to  rigorous  empirical  evaluation. 

A  problem  was  deemed  susceptible  to  rigorous  empirical  evaluation  if 
qualitative  and  quantitative  measures  to  assess  the  impact  of  the  problem  could 
be  determined. 

C6. 

The  researchers  used  the  description  of  the  problem  and  the  potential  solutions 
to  determine  whether  or  not  the  relevant  activities  could  be  manipulated  to 
evaluate  the  problem.  This  depends  on  the  amount  of  changes  required  to 
existing  processes  and  the  perceived  difficulty  in  convincing  study  participants 
to  accept  the  changes. 

In  addition  to  these  general  discussions  of  each  problem  area,  the  participants  were  asked  to 
provide  quantitative  information  on  the  problems.  The  collection  of  this  information  was 
supported  by  a  series  of  questions  with  closed-ended  response  options,  as  in  Table  4.  The 
criterion  Cl  required  the  participants  to  indicate  the  importance  of  the  problem  on  a  scale  of  1  to 
10.  For  consistency,  they  were  not  provided  with  the  opportunity  to  use  alternative  measures  of 
importance. 
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Table  4:  Quantitative  Assessment  -  Standard  questions 


Criterion 

Survey  Questions 

Cl. 

How  important  is  the  problem? 

Please  answer  according  to  your  general  impressions.  Later  questions  will  focus 
on  specific  criteria. 

123456789  10 

Low  Importance  High  Importance 

C2. 

Consider  the  impact  of  the  problem  on  C2  capability  increments  or  (sub-) 
projects  in  the  following  areas.  Give  examples  of  the  impact  and  then  try  to 
quantify  the  impact  (eg  percentage  change). 

Cost  Schedule  Quality 

Interoperability  Client/Developer  relations 

C5. 

What  is  the  time  delay  between  the  start  of  the  activity  involving  the 
problem  and  its  impact? 

<  1  day  (Immediate)  <  1  week  <  1  month 

<  3  months  <  1  year  >  1  year 

Other  (Please  snecifv) 

(eg  event  driven  -  upon  release  etc) 

How  frequently  does  the  activity  involving  the  problem  occur? 

Continual  Every  week  Every  month  Every  year 

Every  release  Every  project 

Other  (Please  specify) 

Five  questions  were  used  to  address  the  criterion  C2  that  concerns  the  impact  of  problem.  Three 
of  these,  cost,  schedule  and  quality,  are  commonly  used  in  the  evaluation  of  projects. 
Interoperability  is  an  aspect  of  quality  that  is  considered  separately  because  it  has  become 
increasingly  important.  The  final  area  is  one  that  is  not  normally  considered,  but  which  can 
affect  not  only  the  outcome  of  a  single  project,  but  the  outcome  of  future  projects.  It  is  the 
interaction  between  Defence  and  Industry.  Alternative  measures,  such  as  user  satisfaction  were 
not  chosen  because  they  are  too  indirect  and  difficult  to  quantify.  However,  not  all  interviewees 
were  able  to  provide  quantitative  information  to  address  the  five  areas  used  to  address  criterion 
C2.  Therefore,  the  participants  were  also  asked  to  provide  additional  qualitative  information 
describing  how  these  five  areas  were  affected. 

Two  questions  were  used  to  address  criterion  C5,  which  concerns  the  time  frames  associated 
with  the  problem.  The  two  questions  concerned  the  time  lag  between  the  inception  and  the 
realisation  of  the  problem,  and  the  frequency  with  which  the  problem  occurs.  Standard,  closed- 
ended,  response  options  were  used  for  these  questions;  although,  the  participants  were  given  the 
option  of  providing  a  response  that  did  not  fit  the  pre-defined  responses. 

The  quantitative  data  was  analysed  using  standard  statistical  techniques.  However,  the  analysis 
was  limited  to  calculating  the  frequency  with  which  the  problems  were  raised  and  the  median  of 
the  response  options,  because  of  the  small  number  of  interviews  conducted.  (Note  that  it  was 
possible  to  use  the  median  (middle)  response  rather  than  the  mode  response  (the  most  common 
response),  because  the  response  options  for  all  of  the  questions  in  Table  4  were  ordered.) 
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4.4  Conduct 

This  section  builds  on  the  summary  of  the  approach  provided  in  Section  3.2.  The  approach  was 
designed  to  reduce  the  likelihood  of  biases  due  to  the  study  participants’  failure  to  recall 
information  or  indecision,  and  to  reduce  the  impact  of  any  biases  introduced  by  the  researchers 
while  recording  and  coding  information. 

Figure  4  shows  the  activities  that  were  undertaken  by  the  study  participants  before,  during  and 
after  the  interview.  Several  of  these  activities  were  included  to  limit  the  biases  that  could  arise  if 
the  participants  failed  to  recall  information. 

The  participants  were  given  the  opportunity  to  prepare  for  the  review  (Preparation).  They  were 
provided  with  a  short  statement  docimienting  the  purpose  of  the  interview  (Appendix  A),  a 
blank  questionnaire  (Appendix  B),  two  worked  examples  of  problems  (Appendix  C),  and  a  copy 
of  the  ICD  process  model  (Figure  1).  It  was  suggested  that  they  prepare  for  the  review  by 
considering  problems  that  they  had  encountered  in  the  ICD  process.  The  participants  were 
generally  given  about  a  week  to  consider,  and  note,  the  problems  they  encounter  in  the  ICD 
process.  This  meant  that  the  participants  did  not  need  to  rely  solely  on  their  ability  to  recall 
information  during  the  interview.  However,  it  should  be  noted  that  the  researchers  had  no 
control  over  the  participants’  preparation. 

The  interview  process  was  also  designed  to  reduce  potential  biases  that  could  arise  if  the 
participants  failed  to  recall  information.  The  researchers  recorded  information  during  all  the 
activities  that  were  conducted  during  the  interview.  The  information  collected  during  the 
Process  Model  Evaluation  stage  was  used  to  crosscheck  the  information  collected  during  the 
Problem  Identification  stage.  For  example,  consider  a  participant  who  is  involved  in  a  particular 
ICD  process,  but  who  does  not  state  any  problems  related  to  that  process.  This  could  occur 
because  of  failure  to  recall  problems  related  to  the  process,  or  because  they  have  not 
encountered  any  problems  with  that  process.  When  the  information  between  the  two  interview 
stages  is  crosschecked,  it  provides  an  opportunity  to  clarify  the  participant’s  comments  and  a 
second  opportunity  for  problems  related  to  the  ICD  process  to  be  identified. 

The  identification  of  problems  was  conducted  separately  to  the  assessment  activity  to  ensure 
that  the  participants  were  not  interrupted  during  the  identification  of  problems.  If  the 
participants  were  interrupted  while  trying  to  identify  problems,  it  is  possible  that  they  would 
have  failed  to  recall  some  problems.  At  natural  breaks  in  the  discussions,  or  approximately  half 
way  through  the  interview,  the  participants  were  asked  to  assess  or  characterise  the  problems 
that  they  had  identified.  This  information  was  recorded  in  an  application  developed  using  MS 
Access.  This  activity  was  driven  by  the  questionnaire  previously  discussed.  The  characterisation 
of  one  problem  area  often  led  to  another  problem  identification  round. 

The  final  steps  in  the  process  (Provide  Additional  Information  and  Check  Recorded 
Information)  were  used  not  only  to  reduce  biases  that  could  be  caused  by  the  participant’s 
failure  to  recall  information  during  the  interview,  but  also  to  reduce  the  impact  of  biases 
introduced  by  the  researchers  during  the  coding  of  the  information.  The  participants  were  given 
copies  of  the  information  recorded  during  the  interview.  To  address  potential  recall  biases,  the 
participants  were  asked  to  complete  missing  information,  and  to  provide  information  about  any 
problems  that  were  not  discussed  during  the  interview  process.  To  address  the  potential  coding 
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biases,  they  are  asked  to  provide  corrections  to  the  recorded  information.  (It  should  be  noted 
that  the  questionnaire  was  also  designed  to  minimise  the  amount  of  coding  required.) 

4.5  Evaluation  of  Materials 

One  final  step  was  taken  to  remove  potential  biases,  such  as  biases  due  to  the  researchers’ 
preconceptions,  from  the  interview  process.  The  first  two  interviews  were  used  to  pilot  the 
questionnaire  and  the  interview  technique.  The  participants  in  these  interviews  were  asked  to 
evaluate  the  questionnaire  and  to  suggest  improvements  to  the  interview  process.  Furthermore, 
two  independent  researchers  studied  the  first  interview  and  highlighted  eighteen  points  that 
could  be  improved,  or  that  required  continued  awareness  from  the  researchers.  In  summary, 
these  areas  were: 

1 .  minor  changes  to  the  interview  pack; 

2.  clearer  introduction  and  identification  of  the  roles  of  the  two  researchers; 

3.  earlier  identification  of  the  participants  role  in  the  ICD  process  diagram; 

4.  the  need  to  supplement  the  database  entry  with  manual  recording  of  a)  initial 
information,  b)  changes  to  the  ICD  process  model,  and  c)  information  about  other 
contacts; 

5.  the  need  to  be  aware  of  the  biases  of  the  interviewee  and  the  researchers,  and  the 
subsequent  need  to  verify  the  information  collected;  and 

6.  the  need  to  be  aware  of  the  difficulty  of  obtaining  some  of  the  requested  information. 

Changes  to  areas  1,  2,  3  and  4  were  made  before  the  remaining  interviews  in  this  study.  Area  5 
was  addressed  by  asking  the  participants  to  focus  on  their  area  of  expertise,  through  the  early 
identification  of  the  source  of  second-hand  information,  and  through  the  requests  for  detailed 
information  and  examples  requested  for  each  problem  area.  Area  6  was  addressed,  as  described 
in  Section  4.3,  by  explicitly  allowing  the  participants  to  provide  examples  as  well  as  quantitative 
information  about  the  detailed  impact  of  problems. 

5.  Application  of  the  Approach 

Sixteen  participants  in  the  ICD  process  were  interviewed.  They  included  the  C3I/IM 
Development  Committee  (the  Gang  of  Seven,  or  the  G7),  three  project  managers,  a  C3I  system 
user,  and  six  other  interviewees.  One  hundred  and  seven  problems  were  recorded  in  the 
database,  relating  to  thirteen  main  problem  areas. 

Four  of  these  problem  areas  were  highlighted  as  areas  for  further  research,  and  examples  of 
problems  and  potential  solutions  were  identified  for  several  of  the  thirteen  problem  areas 
(Henderson  and  Kingston  1999).  One  of  the  main  benefits  of  the  approach  is  that  it  enabled  the 
identification  of  problem  areas  that  are  not  specific  to  a  particular  project.  Thus,  it  provides  a 
more  general  overview  of  the  ICD  process  than  do  “lessons  learned”  reports,  which  usually 
focus  on  a  single  project. 

This  is  not  intended  to  downplay  the  benefits  of  “lessons  learned”  reports.  While  they  often 
contain  subjective,  sometimes  biased,  information,  they  also  provide  valuable,  detailed, 
information  about  specific  activities.  With  this  in  mind,  the  lessons  learned  while  conducting 
the  interviews  are  now  discussed. 
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5.1  Guidelines 

Sections  3  and  4  described  an  approach  to  investigating  the  problems  in  ICD.  This  section 
focuses  on  the  experiences  gained  while  conducting  the  interviews,  both  during  the  evaluation 
interviews,  and  the  primary  interviews. 

During  the  interviews,  the  researchers  had  several  objectives  including  eliciting  information 
from  the  interviewees,  recording  the  information,  and  developing  a  rapport  with  the 
interviewees.  The  researchers  had  to  work  together  to  accomplish  these  objectives.  This  was 
achieved  by  clearly  identifying  the  roles  of  the  researchers.  Initial  roles  were  identified  before 
the  first  (evaluation)  interview.  However,  teamwork  is  a  complicated  process,  so  the  roles  were 
refined  based  on  an  evaluation  of  the  first  interview.  One  researcher  had  prime  responsibility 
for  establishing  a  rapport  with  the  interviewees  and,  for  eliciting  information  on  the  ICD  process 
model  and  associated  problem  domains.  The  other  researcher  had  prime  responsibility  for 
recording  the  information  and  eliciting  information  during  the  evaluation  of  the  problems. 
However,  both  researchers  needed  to  share  the  responsibility  for  achieving  all  three  objectives. 
It  is  recommended  that  researchers  should  share  responsibilities  in  all  interview-based  studies. 
For  example,  while  one  researcher  is  engaged  in  conversation  or  recording  information,  the 
other  researcher  could  try  to  identify  potential  threats  to  the  validity  of  the  information.  A  threat 
to  the  validity  of  the  information  arises  when  uncontrolled  variables  effect  the  information 
collected. 

Three  potential  threats  that  are  associated  with  the  elicitation  of  information  were  identified  in 
the  ICD  problem  identification  study. 

First,  the  study  participants  tended  to  focus  on  problems  that  they  had  recently  encountered  or 
discussed.  Refocussing  the  participants  on  other  problems  required  considerable  care.  The 
preliminary  discussions  on  the  ICD  process,  combined  with  a  general  knowledge  of  the 
acquisition  process,  helped  to  identify  problems  associated  with  less  recent  ICD  activities. 
However,  in  affecting  the  redirection,  the  researchers  needed  to  be  careful  not  to  ask  leading 
questions.  This  was  particularly  true  with  the  quieter  participants.  (Note  however,  that  most 
participants  were  forthcoming  with  information.)  Consequently,  a  debriefing  process  after  each 
interview  that  focuses  on  the  actions  of  the  researchers,  and  the  interpretation  of  the  results,  is 
recommended. 

Second,  it  was  difficult  to  get  additional  information  from  the  participants  after  the  interviews. 
The  researchers  sent  participants  copies  of  the  information  recorded  during  the  interview,  and 
requests  for  additional  information.  However,  in  most  instances,  additional  information  was 
only  obtained  after  several  requests,  and  in  a  few  cases,  important,  missing,  information  was  not 
obtained  even  after  several  requests.  Consequently,  it  is  recommended  that  the  researchers 
consider  the  priorities  of  the  information  of  interest,  and  ensure  that  all  the  highest  priority 
information  is  collected  during  the  interview.  Although  the  demands  on  the  participants’  time 
should  be  minimised,  additional  interviews  should  be  scheduled  if  necessary. 

Third,  there  is  the  potential  for  the  information  to  be  biased  because  of  interpretation  problems. 
This  occurs  when  the  researchers  misinterpret  information,  and  when  participants  identify 
problems  about  which  they  have  only  indirect  knowledge.  To  avoid  this  problem,  the 
researchers  asked  questions  designed  to  clarify  the  nature  of  the  problem.  Some  of  the 
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questions,  such  as  requests  for  examples  and  descriptions  of  the  impact,  were  based  on  the 
questionnaire  discussed  in  Section  4.2.  However,  the  researchers  also  asked  other  questions  on 
an  ad  hoc  basis.  For  example,  when  one  researcher  was  recording  information,  the  other 
researcher  was  asking  questions  designed  to  ensure  that  the  correct  information  was  being 
recorded. 

Information  was  recorded  using  a  computer-based  system,  supplemented  by  a  paper-based 
system.  This  process  proved  to  have  several  benefits.  First,  it  simplified  the  coding  and  analysis 
of  the  information  as  most  of  the  information  was  coded  as  it  was  entered.  Second,  it  provided 
an  opportunity  to  verify  the  coding  and  the  researchers’  interpretations  of  the  information.  The 
study  participants  were  given  regular  opportunities  to  comment  on  the  coding  of  the  information 
-  particularly  after  lengthy  periods  of  discussion.  Third,  it  had  an  unanticipated  benefit.  In  the 
first  (evaluation)  interviews,  the  computer  was  introduced  at  the  start  of  the  interview.  Some  of 
the  participants  stated  that  they  found  this  daunting  and/or  distracting,  so  the  introduction  of  the 
computer  was  delayed  until  the  evaluation  of  the  problem  areas.  This  had  an  added  benefit;  the 
change  seemed  to  stimulate  some  participants.  Having  previously  stated  that  they  could  think  of 
no  more  problem  areas,  several  participants  immediately  identified  other  problem  areas  when 
the  computer  was  introduced,  usually  about  half-way  through  the  interview.  Participants  also 
identified  other  areas  as  they  discussed  detailed  information  about  a  particular  problem  area. 

However,  the  use  of  a  computer  can  also  have  limitations.  The  researcher  responsible  for 
entering  the  information  was  able  to  touch-type  with  reasonable  speed  and  accuracy.  This 
enabled  information  to  be  entered  with  relatively  few  interruptions  and  delays  to  the 
discussions.  (Note  that  the  teaming  arrangements  already  discussed  were  also  necessary  to 
ensure  that  the  use  of  the  computer  did  not  silence  the  participants.)  If  researchers  are  unable  to 
touch-type  then,  subject  to  security  considerations,  alternative  arrangements  -  such  as  the  use  of 
tape-recorders  -  should  be  considered.  However,  researchers  should  be  aware  that  additional 
effort  might  then  be  required  to  verify  the  information  provided,  and  to  ensure  that  the 
researchers  and  the  study  participants  have  a  shared  understanding  of  each  problem  area 
discussed. 


6.  Conclusions 


The  preceding  sections  of  this  report  have  presented  and  discussed  the  design  and  conduct  of  a 
study  that  has  been  used  to  investigate  problems  in  the  ICD  process.  Detailed  results  of  the 
study  are  given  in  (Henderson  and  Kingston  1999).  It  is  hoped  that  the  results  of  this  study  will 
help  to  motivate  process  improvement  activities  in  the  ICD  process.  This  section  discusses  how 
other  researchers  can  benefit  from  the  approach  described  in  this  report. 

Repeated  application  of  the  approach  can  be  used  to  monitor  process  improvement  activities. 
The  results  of  each  study  provide  a  record  of  the  problems  encountered  in  the  ICD  process  at  a 
particular  point  in  time.  As  process  changes  are  implemented,  the  problems  encountered  with 
the  ICD  process  will  change.  Repetition  of  the  approach  could  be  used  to  identify  areas  where 
the  process  improvement  activities  have  been  successful,  and  areas  where  further  changes  are 
required.  However,  changes  in  strategic  guidance  and  changes  in  technology  will  also  affect  the 
acquisition  process.  These  changes  should  be  considered  by  assessing  the  impact  of  process 
improvement  activities.  For  example,  traditional  ‘big  bang’  approaches  were  generally 
satisfactory,  until  the  technological  and  strategic  changes  that  lead  to  Defence’s  acquisition  of 
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software-intensive  C3I  systems.  Future  changes  could  affect  the  suitability  of  ICD  processes. 

Any  assessment  of  process  improvement  activities  needs  to  consider  such  changes. 

In  conducting  these  repetitions,  or  similar  interview-based  studies  in  other  areas,  the  following 

points  should  be  considered. 

•  Evaluate  the  interview  process.  While  this  report  provides  guidelines  and  suggested  roles 
for  participants,  different  research  teams  will  have  different  dynamics  that  need  to  be 
explored  and  evaluated.  Teams  need  to  consider  not  only  how  they  will  work  together  when 
the  process  is  implemented  as  planned,  but  also  how  to  cope  with  deviations  that  arise  when 
interviewing  busy,  sometimes  distracted,  people. 

•  Be  consistent  in  selecting  interview  participants.  It  can  be  tempting  to  select  interview 
participants  in  a  manner  that  biases  the  results  of  the  study.  Potential  participants  should  not 
selected  (or  ignored)  just  because  of  their  familiarity  with  the  research,  their  perceived 
contribution  to  the  study,  or  their  agreement  (or  otherwise)  with  the  researchers  pre¬ 
conceived  ideas.  While  it  might  be  easy  to  adhere  to  this  principle  at  the  start  of  the 
interviews,  the  temptation  increases  if  selected  participants  are  not  available  within  the 
desired  time  period.  For  example,  although  it  extended  the  duration  of  the  study,  the 
complete  Gang  of  Seven  was  interviewed. 

•  Identify  the  information  to  be  collected.  Semi-structured  interviews  based  around  the 
information  to  be  collected  reduce  the  possibility  of  the  researcher  biasing  the  results  of  the 
study  while  still  providing  flexibility.  A  semi-structured  interview  process  can  help  guide 
the  interview  and  identify  the  roles  of  team  members.  It  allows  the  information  from  the 
various  participants  to  be  collated  and  compared,  and  simplifies  the  analysis  of  the 
information. 

•  Consider  using  tool  support.  This  reduces  the  amount  of  coding  required  after  the 
interviews.  Reiteration  of  the  information  as  it  is  recorded  provides  a  valuable  opportunity 
for  verification  of  the  information  -  both  for  checking  the  participant’s  knowledge  of  the 
topic  and  for  ensuring  that  the  coding  does  not  reflect  the  preconceptions  or  biases  of  the 
researchers. 

•  Identify  potential  limitations  early.  The  time  constraints  of  an  interview,  and  the 
knowledge  of  the  participants,  mean  that  it  is  not  always  possible  to  collect  all  of  the  desired 
information  during  each  interview.  Researchers  should  consider  alternative  strategies  that 
can  be  used  if  the  participants  are  unable  or  unwilling  to  provide  information,  and  how  to 
ensure  that  the  most  important  information  is  collected  during  the  review. 
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Appendix  A 

This  appendix  contains  the  flier  used  to  inform  the  study  participants  about  the  purpose  of  the 
study.  It  was  sent  with  a  copy  of  the  diagram  in  Figure  1,  the  questioimaire  in  Appendix  B,  and 
the  sample  problems  in  Appendix  C. 
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Iterative  Capability  Development  Preliminary  Study  - 
Information  for  Participants 

Derek  Henderson  and  Gina  Kingston 


Introduction 

The  aim  of  this  study  is  to  investigate  the  nature  and  severity  of  problems  faced  by  Defence  in  the 
Iterative  Capability  Development  (ICD)  of  Command  and  Control  (C2)  systems.  Iterative  capability 
development  is  a  cyclic  process  where  the  capability  evolves  in  an  evolutionary  way  in  response  to 
changing  operational  requirements  and  new  technology.  It  includes  evolutionary  acquisition  but  also 
focuses  on  other  elements  that  need  to  be  considered  in  the  development  of  effective  capability,  such  as 
high  level  planning,  approval  processes,  research  and  development,  and  support. 

Key  stakeholders  will  be  interviewed  and  their  views  sought  on  the  major  ICD  issues.  Interviews  will 
be  structured  around  a  questionnaire  developed  by  the  researchers.  Quantitative  evidence  relating  to 
problems  will  be  collected  and  stored  for  subsequent  retrieval  and  analysis.  This  evidence  will  be  used 
to  develop  empirical  studies  of  specific  problems  in  ICD  that  stakeholders  agree  are  important.  The 
output  from  the  empirical  studies  will  be  a  series  of  credible  recommendations  for  improvements  to 
ICD  processes,  which  will  ultimately  result  in  the  acquisition  of  more  effective  C2  systems. 


Study  Objectives 

Study  Benefits  (to  participants  and 
Defence) 

1 .  Identification  of  ICD  problems. 

2.  Characterisation  of  the  problems  in  terms  of  their 
nature,  importance  and  impact. 

3.  Comparison  of  different  stakeholder’s  perceptions 
of  problems. 

4.  Development  of  a  high  level  ICD  process  model. 

5.  Identification  of  a  small  number  of  critical 
problems,  which  can  addressed  in  later,  detailed 
empirical  studies.  The  objective  of  these  studies 
will  be  to  uncover  objective  evidence  to  use  as  the 
basis  of  recommendations  for  process 
improvement. 

Short  term  benefits 

1.  A  better  understanding  of  ICD  processes  for  the 
development  of  C2  systems. 

2.  An  understanding  (and  visibility)  of  the  various 
problems  in  ICD  processes. 

3.  Potential  solutions  to  problems  in  ICD  processes. 

Longer  term  benefits 

1.  Credible  recommendations  based  on  objective 
evidence,  to  critical  problems  in  the  ICD  process. 

The  Researchers 

The  Participants 

Derek  Henderson  and  Gina  Kingston  work  in  the 
Software  Systems  Engineering  (SSE)  Group  of 
Information  Technology  Division  at  DSTO.  SSE  group 
has  provided  advice  on  a  consultancy  basis  to  a  number 
of  projects  including  JCSE,  AEW&C,  Nulka,  Starlight, 
and  HFMOD.  Derek  has  a  wide  experience  in  Defence 
and  has  recently  conducted  research  into  requirement 
specifications  and  Evolutionary  Acquisition.  Gina  is 
currently  studying  Joint  Reviews  for  her  PhD  and  has  a 
background  in  empirical  methods. 

Participants  have  been  selected  who  have  a  key 

involvement  in  the  ICD  process. 

Preparation.  As  a  participant  you  should: 

1 .  Become  familiar  with  the  ICD  model  provided  and 
understand  how  your  role  fits  within  the  model. 

2.  Identify  problems  within  the  ICD  process  that  you 
have  encountered  in  your  role.  Completed  sample 
problems  are  provided  as  examples. 

Further  reading 

The  following  documents  are  provided  in  this  information  pack  as  background  to  the  study: 

1.  Iterative  Capability  Development  of  Command  and  Control  Systems  -  A  Research  Proposal  -  This 
paper  provides  an  overview  of  ICD  and  briefly  describes  the  proposed  research  program. 

2.  A  questionnaire.  Participants  do  not  have  to  complete  questionnaires;  information  will  be  entered 
directly  into  a  database  by  a  researcher  during  the  interview. 


3.  A  Model  of  the  ICD  process  -  This  model  represents  our  current  understanding  of  the  ICD 
processes  and  will  be  explained  during  the  interview.  We  intend  to  evolve  this  model  as  we  gain 
feedback  from  participants. 


Confacf  details: 

Derek  Henderson 

Gina  Kingston 

Email:  Derek.Henderson®dsto.defence.aov.au 

Email:  Gina.lCingston(S!dsto.defence. aov.au 

Tel:  (08)  8259  7667 

Tel:  (08)8259  6611 

Address:  ITD/DSTO, 

Address:  ITD/DSTO, 

PO  Box  1500,  Salisbury, 

PO  Box  1500,  Salisbury, 

South  Australia  5108 

South  Australia  5108 
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Appendix  B;  Questionnaire 

This  appendix  contains  the  questionnaire  used  in  the  study.  The  questionnaire  was  used 
primarily  as  a  guide  to  the  information  required.  The  participants  were  not  required  to  complete 
the  questionnaire.  Instead,  the  information  was  entered  directly  into  a  database  during  the 
interviews.  However,  the  study  participants  were  left  with  a  copy  of  the  questionnaire  in  case 
they  wished  to  provide  information  on  additional  problems  not  covered  during  the  interview. 
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Interviewee; 
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Id: 

Parent: 


Problem  Description: 


Impact ; 

Potential  Solutions ; 

Part  B:  Characteristics 


How  important  is  the  probiem? 

Please  answer  according  to  your  general  impressions.  Later  questions  will  focus  on  specific  criteria. 

123456789  10 

Low  Importance  High  Importance 


ICD  Preliminary  Study:  Problem  Elaboration  Questionnaire 


Part  A:  Descriptions 


Example; 


What  is  the  time  delay  between  the  start  of  the  activity  involving  the  problem  and  its  impact? 

<  1  day  <  1  week  <  1  month 

<  3  months  <  1  year  >  1  year 

How  frequently  does  the  activity  involving  the  problem  occur? 

Every  week  Every  month  Every  year  Every  project 
Other _  (Please  specify) 

Who  has  authority  to  change  the  activity  involving  the  problem? 


Consider  the  impact  of  the  problem  on  the  how  C2  capability  increments  or  (sub-)  projects  in  the 
following  areas.  Give  examples  of  the  impact  and  then  try  to  quantify  the  impact. 


Area  Example  Percentage 

_ _ _  Change 

Cost 
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Appendix  C:  Sample  Problems 

This  appendix  contains  the  worked  examples  distributed  to  the  study  participants  before  the 
interviews.  It  was  sent  with  the  flier  in  Appendix  A,  a  copy  of  the  diagram  in  Figure  1,  and  the 
questionnaire  in  Appendix  B. 
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ICD  Preliminary  Study:  Problem  Elaboration  Questionnaire 


Part  A:  Descriptions 


Part  B:  Characteristics 


How  important  is  the  problem? 

Please  answer  according  to  your  general  impressions.  Later  questions  v^focus  on  specific  criteria. 

1  23  456  7(^9  10 

Low  Importance  High  Importance 

What  is  the  time  delay  between  the  start  of  the  activity  involving  the  problem  and  its  impact? 

<  1  day  <  1  week  <  1  month 

<  3  months  <  1  year  >  1  year 

How  frequently  does  the  activity  involving  the  problem  occur? 

Every  week  Every  month  Every  year  Every  project 

Other  _  (Please  specify) 


Who  has  authority  to  change  the  activity  involving  the  problem? 


Consider  the  impact  of  the  problem  on  the  how  C2  capability  increments  or  (sub-)  projects  in  the 
following  areas.  Give  examples  of  the  impact  and  then  try  to  quantify  the  impact 
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ICD  Preliminary  Study:  Problem  Elaboration  Questionnaire 


Part  A:  Descriptions 


Part  B:  Characteristics 


How  important  is  tbe  problem? 

Please  answer  according  to  your  general  impressions.  Later  questions  will  focus  on  specific  criteria. 

1  2  3  4  5  6  7^  910 

Low  Inqjortance  High  Importance 

What  is  the  time  delay  between  the  start  of  the  activity  involving  the  problem  and  its  impact? 

<  1  day  <  1  week  <  1  month 

<  3  months  <  1  year  >  1  year 


How  frequently  does  the  activity  involving  the  problem  occur? 

Every  week  Every  month  Every  year  Every  project 

Other _ (Please  specify) 


Who  has  authority  to  change  the  activity  involving  the  problem? 

■VV'*- 


Consider  the  impact  of  the  problem  on  the  how  C2  capability  increments  or  (sub-)  projects  in  the 
following  areas.  Give  e^camples  of  the  impact  and  then  try  to  quantify  the  impact 


Example 


Schedule 


Quality 


Interoperability 


Client-Developer 

Relations 
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